{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Validation Playground\n",
    "\n",
    "**Watch** a [short tutorial video](https://greatexpectations.io/videos/getting_started/integrate_expectations) or **read** [the written tutorial](https://docs.greatexpectations.io/en/latest/tutorials/validate_data.html?utm_source=notebook&utm_medium=validate_data)\n",
    "\n",
    "#### This notebook assumes that you created at least one expectation suite in your project.\n",
    "#### Here you will learn how to validate data loaded into a PySpark DataFrame against an expectation suite.\n",
    "\n",
    "\n",
    "We'd love it if you **reach out for help on** the [**Great Expectations Slack Channel**](https://greatexpectations.io/slack)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import great_expectations as ge\n",
    "import great_expectations.jupyter_ux\n",
    "from great_expectations.datasource.types import BatchKwargs\n",
    "import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Get a DataContext\n",
    "This represents your **project** that you just created using `great_expectations init`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context = ge.data_context.DataContext()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Choose an Expectation Suite\n",
    "\n",
    "List expectation suites that you created in your project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context.list_expectation_suite_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expectation_suite_name =  # TODO: set to a name from the list above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Load a batch of data you want to validate\n",
    "\n",
    "To learn more about `get_batch`, see [this tutorial](https://docs.greatexpectations.io/en/latest/tutorials/validate_data.html?utm_source=notebook&utm_medium=validate_data#load-a-batch-of-data-to-validate)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list datasources of the type SparkDFDatasource in your project\n",
    "[datasource['name'] for datasource in context.list_datasources() if datasource['class_name'] == 'SparkDFDatasource']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasource_name = # TODO: set to a datasource name from above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you would like to validate a file on a filesystem:\n",
    "batch_kwargs = {'path': \"YOUR_FILE_PATH\", 'datasource': datasource_name}\n",
    "# To customize how Spark reads the file, you can add options under reader_options key in batch_kwargs (e.g., header='true')\n",
    "\n",
    "# If you already loaded the data into a PySpark Data Frame:\n",
    "batch_kwargs = {'dataset': \"YOUR_DATAFRAME\", 'datasource': datasource_name}\n",
    "\n",
    "\n",
    "batch = context.get_batch(batch_kwargs, expectation_suite_name)\n",
    "batch.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Validate the batch with Validation Operators\n",
    "\n",
    "`Validation Operators` provide a convenient way to bundle the validation of\n",
    "multiple expectation suites and the actions that should be taken after validation.\n",
    "\n",
    "When deploying Great Expectations in a **real data pipeline, you will typically discover these needs**:\n",
    "\n",
    "* validating a group of batches that are logically related\n",
    "* validating a batch against several expectation suites such as using a tiered pattern like `warning` and `failure`\n",
    "* doing something with the validation results (e.g., saving them for a later review, sending notifications in case of failures, etc.).\n",
    "\n",
    "[Read more about Validation Operators in the tutorial](https://docs.greatexpectations.io/en/latest/tutorials/validate_data.html?utm_source=notebook&utm_medium=validate_data#save-validation-results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is an example of invoking a validation operator that is configured by default in the great_expectations.yml file\n",
    "\n",
    "\"\"\"\n",
    "Create a run_id. The run_id must be of type RunIdentifier, with optional run_name and run_time instantiation\n",
    "arguments (or a dictionary with these keys). The run_name can be any string (this could come from your pipeline\n",
    "runner, e.g. Airflow run id). The run_time can be either a dateutil parsable string or a datetime object.\n",
    "Note - any provided datetime will be assumed to be a UTC time. If no instantiation arguments are given, run_name will\n",
    "be None and run_time will default to the current UTC datetime.\n",
    "\"\"\"\n",
    "\n",
    "run_id = {\n",
    "  \"run_name\": \"some_string_that_uniquely_identifies_this_run\",  # insert your own run_name here\n",
    "  \"run_time\": datetime.datetime.now(datetime.timezone.utc)\n",
    "}\n",
    "\n",
    "results = context.run_validation_operator(\n",
    "    \"action_list_operator\",\n",
    "    assets_to_validate=[batch],\n",
    "    run_id=run_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. View the Validation Results in Data Docs\n",
    "\n",
    "Let's now build and look at your Data Docs. These will now include an **data quality report** built from the `ValidationResults` you just created that helps you communicate about your data with both machines and humans.\n",
    "\n",
    "[Read more about Data Docs in the tutorial](https://docs.greatexpectations.io/en/latest/tutorials/validate_data.html?utm_source=notebook&utm_medium=validate_data#view-the-validation-results-in-data-docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context.open_data_docs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations! You ran Validations!\n",
    "\n",
    "## Next steps:\n",
    "\n",
    "### 1. Read about the typical workflow with Great Expectations:\n",
    "\n",
    "[typical workflow](https://docs.greatexpectations.io/en/latest/getting_started/typical_workflow.html?utm_source=notebook&utm_medium=validate_data#view-the-validation-results-in-data-docs)\n",
    "\n",
    "### 2. Explore the documentation & community\n",
    "\n",
    "You are now among the elite data professionals who know how to build robust descriptions of your data and protections for pipelines and machine learning models. Join the [**Great Expectations Slack Channel**](https://greatexpectations.io/slack) to see how others are wielding these superpowers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
